Uncertainty-Aware Image Captioning

It is well believed that the higher uncertainty in a word of caption, more inter-correlated context information required to determine it. However, current image captioning methods usually consider generation all words sentence sequentially and equally. In this paper, we propose an uncertainty-aware framework, which parallelly iteratively operates insertion discontinuous candidate between existing from easy difficult until converged. We hypothesize high-uncertainty need prior make correct decision should be produced at later stage. The resulting non-autoregressive hierarchy makes caption explainable intuitive. Specifically, utilize image-conditioned bag-of-word model measure apply dynamic programming algorithm construct training pairs. During inference, devise uncertainty-adaptive parallel beam search technique yields empirically logarithmic time complexity. Extensive experiments on MS COCO benchmark reveal our approach outperforms strong baseline related both quality as decoding speed.